An image capturing apparatus such as a digital still camera or digital video camera guides and forms an image represented by light coming from an object onto a CCD or CMOS sensor as an image capturing device via an imaging optical system including lenses. The image capturing device converts the received light into an electrical signal. Processes such as analog-to-digital (A/D) conversion and demosaicing, which convert an electrical signal into image data, are applied to this electrical signal to obtain a capturing image.
Since light that reaches the image capturing device passes through the imaging optical system, the image quality of the capturing image is influenced by the imaging optical system. For example, when a high-performance lens is used, a capturing image having a high resolution up to its peripheral region can be obtained. Conversely, when an inexpensive low-performance lens is used, the resolution of especially a peripheral region of a capturing image drops considerably.
For example, when an image of starry sky is to be captured, an image captured using a high-performance lens includes an image of each star as a point image. Each star in an image captured using a low-performance lens, however, is not a point image but is blurred. When an image of a person is to be captured, if a high-performance lens is used, an image that records details of hair can be obtained. However, if a low-performance lens is used, an image in which hair is blurred is obtained. That is, when a low-performance lens is used, an image with low definition is obtained.
Such blur is generated even in an in-focus state and depends on the characteristics of the imaging optical system. In other words, capturing images have different resolutions depending on the performances of lenses even in an in-focus state.
A method of correcting any blur of an image caused by an imaging optical system by applying an image process to a capturing image is known. This method applies the image process based on the blur characteristics of the imaging optical system, which are converted in advance into data, thereby correcting any blur of an image caused by the imaging optical system.
In order to convert the blur characteristics of the imaging optical system into data, for example, a method of using a point spread function (PSF) is available. The PSF represents how one point of an object is blurred. For example, when an image of a light-emitting member having a very small volume (point light source) is captured via an imaging optical system in a dark environment, if an ideal imaging optical system is used, light forms a point on a surface (light-receiving surface) of an image capturing device. However, when an imaging optical system which suffers a large blur is used in the same environment, light does not form a point on the light-receiving surface, and has a certain spread. That is, the two-dimensional distribution of light on the light-receiving surface corresponds to the PSF of that imaging optical system. Upon acquiring the PSF of an imaging optical system in practice, an image of an object like a point light source need not be captured. For example, the PSF can be calculated from an image obtained by capturing an image of a chart having black and white edges by a calculation method corresponding to the chart.
As a method of correcting an image blur based on the PSF, a method using an inverse filter is known. Assume that an image of a point light source is captured in a dark environment for the sake of descriptive convenience. Light emanating from the point light source forms a distribution of light having a certain spread on the light-receiving surface. The light is converted into an electrical signal by an image capturing device. When this electrical signal is converted into image data, a digital image obtained by capturing the image of the point light source is obtained. In an image captured using an imaging optical system having a blur, not only a pixel corresponding to the point light source has a nonzero significant pixel value, but also pixels around that pixel have significant pixel values which are close to zero. An image process which converts this image into that having a significant pixel value at nearly one point is inverse filtering. An inverse filter can obtain an image as if it were captured by an imaging optical system with a minimum blur.
The point light source has been exemplified for the sake of descriptive convenience. If light coming from an object is considered as a set of a large number of point light sources, blurs of light components emerging from or reflected by respective portions of the object can be eliminated, and a less-blurred image can be obtained even for an object other than a point light source.
A practical configuration method of an inverse filter will be described below. Let f(x, y) be a capturing image captured using an ideal imaging optical system free from any blur. (x, y) represent a two-dimensional pixel position on the image, and f(x, y) represents the pixel value of a pixel (x, y). On the other hand, let g(x, y) be a capturing image captured using an imaging optical system having a blur. Let h(x, y) be the PSF of the imaging optical system having a blur. Then, f, g, and h meet:g(x, y)=h(x,y)*f(x,y)  (1)where * represents a convolution operation.
Image blur correction (to be referred to as blur correction hereinafter) amounts to estimating f captured by a blur-free imaging optical system from the image g captured using the imaging optical system having a blur and h as the PSF of that imaging optical system. By computing the Fourier transform of the above equation to obtain a presentation format in the spatial frequency domain, that equation is expressed by the format of products for respective frequencies like:G(u, v)=H(u, v)·F(u, v)  (2)where H is an optical transfer function (OTF) as the Fourier transform of the PSF,
u is the spatial frequency in the x-direction,
v is the spatial frequency in the y-direction,
G is the Fourier transform of g, and
F is the Fourier transform of f.
In order to obtain the blur-free capturing image f from the blurred capturing image g, both the sides of equation (2) can be divided by H like:G(u, v)/H(u, v)=F(u, v)  (3)
By computing the inverse Fourier transform of F(u, v) obtained by equation (3) to restore it to a real space, a blur-free image f(x, y) can be obtained.
Letting R be the inverse Fourier transform of 1/H, a blur-free image is obtained by making a convolution on a real space like:g(x, y)* R(x, y)=f(x, y)  (4)
R(x, y) in equation (4) is called an inverse filter. In practice, since a division by a divisor=0 is generated at a frequency (u, v) at which H(u, v)=0, the inverse filter R(x, y) requires a slight modification.
Normally, the value of the OTF becomes smaller with increasing frequency, and the value of the inverse filter as the reciprocal of the OTF becomes larger with increasing frequency. Therefore, when the convolution of a capturing image is made using an inverse filter, high-frequency components of the capturing image are emphasized to consequently emphasize noise components (noise components are normally high-frequency components) included in the capturing image. Hence, a method of giving characteristics which do not so emphasize high-frequency components compared to the inverse filter by modifying R(x, y) is known. As a filter which does not so emphasize high-frequency components in consideration of noise, a Wiener filter is popular.
In this way, a blur cannot be perfectly removed due to deviation from ideal conditions, that is, the presence of noise included in the capturing image, that of the frequency at which the OTF=0, and so forth. However, a blur can be reduced by the aforementioned process. Note that a filter used in blur correction such as an inverse filter and Wiener filter will be collectively referred to as a “recovery filter” hereinafter. As described above, the recovery filter is characterized by executing the image process using the PSF of the imaging optical system.
A color image typically has pixel values of RGB three colors per pixel. When blur correction is individually applied to images of R, G, and B planes, a blur of a color image can be reduced. Upon execution of the blur correction for each plane, the blur characteristics of the imaging optical system are different for respective colors, and recovery filters are used in correspondence with colors.
When an image of a three-dimensional object is captured, an object image corresponding to a position in front of or behind an in-focus position is blurred compared to an object image that matches the in-focus position. As can be seen from this phenomenon, the PSF which represents a blur of the imaging optical system varies depending on the object distance. Normally, when an object deviates to a position in front of or behind the in-focus position, the PSF changes in correspondence with the spread of the light distribution on the light-receiving surface.
The aforementioned blur correction uses the PSF corresponding to the in-focus position. Although the application of use of the blur correction is not limited, it is often the case that the blur correction is executed under the incentive that a blur of the imaging optical system, which occurs even in an in-focus state, is reduced to obtain a sharper image. When a blur is corrected using the PSF corresponding to the in-focus position, an optimal blur correction effect for an object image at the in-focus position can be obtained. Therefore, under the above incentive, it is optimal to apply blur correction using the PSF corresponding to the in-focus position. However, an image of a three-dimensional object includes an out-of-focus object image. An object (or an object image) which deviates from the in-focus position will be referred to as a “defocus object” hereinafter. When a blur is to be corrected over the entire frame of the capturing image, this defocus object also undergoes the blur correction.
FIG. 1 is a view for explaining artifacts generated upon application of the blur correction to a defocus object.
When a blur of an image obtained by capturing an object 11 (e.g., a person) by an image capturing apparatus 14 is to be corrected, a blur of an object image at an in-focus position 12 is satisfactorily corrected, and a sharp image can be obtained. However, as for an object image at a position 13 which deviates from the in-focus position 12, the blur characteristics of an imaging optical system are different from those at the in-focus position 12. When the same blur correction as that at the in-focus position 12 is applied to the object image at the position 13, false colors are generated at the contour portions of object images at the position 13. In the example shown in FIG. 1, green lines (false colors 15) are generated on the arms of an object image 16 to fringe that object image 16, thus generating artifacts.